1. Field of the Invention
The present invention relates to a multi-layer perceptron circuit device which is widely used in various fields such as recognition problems for letters and voices, optimal control problem for robot motion, general process control problem, and neuro-computers.
2. Description of the Background Art
A perceptron is a neural network invented by Rosenblatt and others in 1940's, which has a multi-layer structure. In general, when a number of layers in this multi-layer structure is large, it is called a multi-layer perceptron.
In general, the multi-layer perceptron has a configuration shown in FIG. 1 which resembles neuron structure in the human brain, for which various mathematical analysis has already been made with regards its information processing capability. For instance, there is a theorem by Kolmogorov which states that "A perceptron having n(2n+1) neurons can express an arbitrary continuous function with n variables". The parameters of the perceptron are given by combination weights of the synapse connections between neurons, and self-governing or adaptive type network can be constructed by updating these parameters according to partial differential equations called the learning equations. More recently, D. E. Rumelhart and J. L. McClelland developed the Backward Error Propagation Algorithm (BEP) in which these parameters can be determined by the method of steepest descent.
Referring now to FIG. 1, a conventional multi-layer perceptron will be described in some detail.
FIG. 1 shows a case of three-layer perceptron comprising the first layer called an input layer which generally contains n.sub.s neurons S.sub.i (i=1, 2, . . . , n.sub.s), the second layer called an intermediate layer which generally contains n.sub.A neurons A.sub.i (i=1, 2, . . . , n.sub.A), and the third layer called an output layer which generally contains n.sub.R neurons R.sub.i (i=1, 2, . . . , n.sub.R). For the sake of simplicity, the FIG. 1 shows the case in which n.sub.S =n.sub.A =n.sub.R =3. Also, in FIG. 1, the combination weights for the synapse connections are designated as RS.sub.i j and RA.sub.i j (i, j=1, 2, 3). In FIG. 1, signals are propagated from left to right, and an input signal and an output signal of each neuron have in general a mapping relation of a non-linear function having a monotonous change such as a logistic function. Also, the input signals for the neurons of the input layer are assumed to be given externally in forms resembling sensing organ stimulating signals, and each of the neurons A.sub.i and R.sub.i of the intermediate layer and the output layer receives a sum of signals coming in with each signal being weighted by the combination weights RS.sub.i j and RA.sub.i j associated with each synapse connection. In other words: ##EQU1##
Next, the Backward Error Propagation algorithm will briefly be reviewed.
The Backward Error Propagation algorithm is a learning algorithm for a case in which a teacher signal is given to the output layer, in which weights are updated according to the following equations: EQU R.sub.ij (t+.DELTA.t)=.alpha.R.sub.ij (t)+.eta..epsilon..sub.j f(y.sub.i (t)) (3)
where f is a mapping from an input signal y.sub.i (t) to an output signal, R.sub.i j represents either RS.sub.i j or RA.sub.i j, .alpha. is called a forgetting factor for which a value 1 corresponds to a case with no forgetting and a value less than 1 corresponds to a case with forgetting with a smaller value indicating faster forgetting, .eta. is called a relaxation factor which determines relaxation in the transition according to the above eq.(3) of the weights to the final values, and .epsilon..sub.j is a signal containing error information which will be given by: EQU .epsilon..sub.j =(T.sub.j -f(y.sub.j (t)))f'(y.sub.j (t)) (4)
for a case in which the neuron is located in the output layer, and where T.sub.j is a teacher signal and a prime indicates a differential, and: ##EQU2## for a case in which the neuron is not located in the output layer.
Now, such a conventional multi-layer perceptron and learning equation for its parameters such as the Backward Error Propagation algorithm just described has been practically implemented, such that the multi-layer perceptron is usually represented only as diagrammatically as FIG. 1 or by a breadboarded circuit at best, and the algorithm is usually checked by either a theoretical calculations or simulation experiments.
In practically implementing the multi-layer perceptron by integrated circuits, there are several problems to be overcome, such as the following.
First of all, there is a problem concerning the arrangement and the wiring of the multi-layer perceptron diagrammatically shown in FIG. 1. Namely, a number of wirings required between two layers containing n neurons is n.sup.2, so that as n increases more sophisticated arrangement and wiring are required in order to avoid undesirable situation such as overlapping of wirings. Moreover, since an integrated circuit can accommodate only a limited number of elements within, it is preferably for each integrated circuit to be easily extendable by connecting other integrated circuits.
Secondly, the possible practical implementation of the Backward Error Propagation algorithm has not been achieved, so that although the use of the method of steepest descent is advocated in this algorithm, how to break this algorithm down to practically implementable blocks have to be worked out.
Thirdly, after the multi-layer perceptron circuit device is constructed, there still is a problem of how to activate an entire device uniformly.